1. Field of the Invention
The present invention relates to an image processing method and apparatus which sense an object (to be sensed) by using a plurality of image sensing systems or a single image sensing system and reproduce a three- or two-dimensional image from the sensed image.
2. Description of the Related Art
Distance measuring devices for measuring the distance to an object to be photographed can be roughly classified into distance measuring devices based on active methods of irradiating some kind of energy onto the object and distance measuring devices based on passive methods of analyzing an image signal sensed by a photoelectric conversion element or the like.
Active methods which are widely used are: a method of irradiating a wave, e.g., a radio wave, an ultrasonic wave, or light, onto an object and measuring the distance to the object on the basis of the propagation time required for the irradiated wave to return upon reflection; and a method (e.g., Moiré topography) of obtaining the shape of an object to be photographed from the geometric distortion of an artificial pattern formed on the object surface by projecting a regular pattern from an appropriate light source.
Passive methods are: a method of measuring a distance, on the basis of the principle of trigonometrical survey, from the positional relationship between corresponding points of images obtained by using a plurality of image sensors; and a method of obtaining the three-dimensional structure of an object to be photographed from the geometric distortion of a pattern on the object surface. In addition, attempts have been made to use a method of obtaining the distance to an object on the basis of images sensed by a plurality of image sensing processing systems having different focal lengths.
As described above, in various conventional methods, the three-dimensional shape of an object is measured to extract three-dimensional information, and a three-dimensional image is reproduced on the basis of the three-dimensional information.
In general, in order to reproduce images, sensed image data and images viewed from predetermined viewpoints and formed by interpolation on the basis of sensed data are stored in advance. In reproducing images, images are selected from the stored images to be displayed.
Several problems are posed in realizing the above various problems.
First of all, there are problems associated with extraction of three-dimensional information required to generate a reproduced image. More specifically, in the above conventional methods, since energy such as an ultrasonic wave or a laser beam must be irradiated on an object to detect the reflecting object, measurement cannot be performed for any object which absorbs the energy. In addition, a strong laser beam cannot be irradiated on a human body in consideration of adverse effects on the human body. That is, the above active methods are poor in versatility. Furthermore, the method of projecting a regular pattern is basically a method used to obtain the relative three-dimensional information of an object so that an absolute distance until the object can not be acquired.
Of the passive methods, the method of using a plurality of image sensors requires a plurality of image sensing elements and image sensing systems and hence is not suitable for a compact, inexpensive distance measuring device. In the method of extracting three-dimensional information from the geometric distortion of a pattern on an object surface, the characteristics of the geometric pattern must be known in advance, and obtained information is only relative distance information. For this reason, the application range of this method is very narrow. In addition, a distance measuring device based on a focus error with respect to an image requires a bright lens with a small depth of focus and a large numerical aperture to perform accurate distance measurement. For this reason, sufficient resolution cannot be obtained by using a general TV camera lens.
The above problems are posed in measuring the three-dimensional shape of an object or an object to be photographed. Secondly, there is no conventional apparatus which reproduces an image viewed from an arbitrary viewpoint position and an arbitrary viewpoint direction from images obtained by sensing an object to be photographed. For example, a conventional video apparatus can only reproduce an image under the same condition as that in a recording operation. This will be described with reference to FIG. 1. When a photographing operation is performed by a general camera, an image is obtained in accordance with the TV frame (field) frequency, as shown in FIG. 1. In reproducing this image, forward reproduction is performed from the position denoted by reference numeral 181 in FIG. 1, or reverse reproduction is performed from the position denoted by reference numeral 182. An image reproduction order cannot be set in accordance with a designated order based on an arbitrary viewpoint position and an arbitrary position of line of sight.
As described above, since many image data including many interpolated image data must be held to reproduce an image, a large-capacity memory is needed. In addition, a much retrieval time is required to pick up one of image data which has been stored in the memory. Owing to the limited memory capacity, it is not practical to retain all images viewed from various viewpoints.